AN EMPIRICAL EXAMINATION OF
ANALYSIS OF COVARIANCE WITH AND WITHOUT
PORTER'S ADJUSTMENT FOR A FALLIBLE COVARIATE
By
JAMES EDWIN McLEAN
A DISSERTATION PRESENTED TO THE GRADUATE
COUNCIL OF THE UNIVERSITY OF FLORIDA IN PARTIAL
FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY
UNIVERSITY OF FLORIDA
1974
iS/«S'.Z?;f/^0R,0A
3 1262 08552 7488
DEDICATION
I dedicate this study to the late Dr. Charles M. Bridges, Jr.,
my teacher, advisor and friend.
ACKNOWLEDGMENTS
I wish to express my grateful appreciation to my committee members
for their guidance in this study. Dr. William B. Ware, my chairman,
suggested the topic and was a major influence in its development. Dr.
James T. McClave-was a constant source of assistance, both theoretically
and editorially. Drs. Vynce Hines, William Mendenhall, and P. V. Rao
all provided direction along with many helpful suggestions. This study
could not have been completed without the spirit of cooperation which I
encountered between members of the two departments involved.
The late Dr. Charles M. Bridges, Jr. first encouraged me to enroll
in the program and served as my chairman until his death. His counseling
and guidance were a major reason for my successful completion of graduate
study.
The searching questions of my fellow students led to several
worthwhile modifications and I wish to express my appreciation to them.
I also wish to thank my wife, Sharon, who for the last five years,
has held two jobs (wife and medical technologist), so that I might
further my education. She has also been most understanding about the
many hours spent away from home.
m
TABLE OF CONTENTS
Page
ACKNOWLEDGMENTS iii
LIST OF TABLES . vi
ABSTRACT . .viii
CHAPTER
I. INTRODUCTION 1
The Problem of Comparing Groups of Differing Abilities .... 1
Methods of Comparing Groups of Differing Abilities ...... 3
Limitations Used in this Study 4
Procedures . . 5
Relevance of the Study 7
Organization of the Dissertation .... 7
II. REVIEW OF THE RELATED LITERATURE 8
Historical Review of the Problem 8
Analysis of Covariance 10
Analysis of Covariance with Porter's Adjustment
for a Fallible Covariate . . . . . . . . . . ... . . . .11
Summary. ............ 14
III. PROCEDURES .16
The Model 16
Selecting the Reliabilities. . . . . . . . . . . . 17
The Regression of Y on X .18
Selecting Means .20
Generation of Random Normal Deviates with Specified
Means and Variances 26
Analysis for Comparing the Selected Methods .26
IV. RESULTS 29
V. DISCUSSION 41
Comparison of the Two Methods of Analysis. . 41
Factors that Affect Alpha and Beta 42
TV
Page
Predicting Alpha and Power . ;.......... 43
A Direction for Future Research. 46
VI. SUMMARY. . . . 47
APPENDIX ... : . . . . . . ... . ... ..... .49
Fortran Program Used to Perform Analysis 50
BIBLIOGRAPHY 56
BIOGRAPHICAL SKETCH. . 59
LIST OF TABLES
Table Page
1. CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN
THE RELIABILITIES FOR BOTH GROUPS WERE EOUAL. ..... 24
2. CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN
THE RELIABILITIES FOR BOTH GROUPS WERE UNEQUAL. . . . .25
3. FACTORIAL DESIGN FOR STANDARD ANALYSIS OF COVARIANCE
WHEN MEAN GAIN OF GAIN GROUP WAS ZERO AND CRITERION
VARIABLES ARE MONTE CARLO GENERATED ALPHA VALUES. ... 28
4. FRACTION OF SIGNIFICANT F'S AND NUMBER OF TIMES F^
EXCEEDS F WHERE THERE WAS NO GAIN IN EITHER GROUP
AND THE PRETEST RELIABILITIES WERE EQUAL . 32
5. FRACTION OF SIGNIFICANT F'S AND NUMBER OF TIMES F^
EXCEEDS fo WHERE THERE WAS NO GAIN IN EITHER GROUP
AND THE PRETEST RELIABILITIES WERE NOT EQUAL 33
6. FRACTION OF SIGNIFICANT F'S AND NUMBER OF TIMES Fg
EXCEEDS L, WHERE THERE WAS GAIN IN THE GAIN GROUP
AND THE PRETEST. RELIABILITIES WERE EQUAL. ....... 34
7. FRACTION OF SIGNIFICANT I'S AND NUMBER OF TIMES fs
EXCEEDS Fp WHERE THERE WAS NO GAIN IN THE GAIN
GROUP AND THE PRETEST RELIABILITIES WERE NOT
EQUAL , 35
8. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS
THE STANDARD ANALYSIS OF COVARIANCE AS THE CRITERION
VARIABLES 36
9. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS
FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT
AS THE CRITERION VARIABLES. ....... . 37
10. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS
FROM THE STANDARD ANALYSIS OF COVARIANCE AS THE
CRITERION VARIABLES ..... 38
11. ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS
FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT
AS THE CRITERION VARIABLES . .... . . .39,
VI
Table Page
12. COEFFICIENTS FOR THE LINEAR CONTRASTS USED WHEN RELIA-
BILITY WAS SIGNIFICANT IN THE ANALYSIS OF VARIANCE. . . 40
13. NINTY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE' EXPECTED
VALUES OF ALPHAS UNDER SPECIFIED CONDITIONS 44
14. NINTY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE EXPECTED
VALUES OF POWERS UNDER SPECIFIED CONDITIONS . . . . . .45
vn
Abstract of Dissertation Presented to the
Graduate Council of the University of Florida in Partial
Fulfillment of the Requirements for the Degree of Doctor of Philosophy
AN EMPIRICAL EXAMINATION OF
ANALYSIS OF COVARIANCE WITH AND WITHOUT
PORTER'S ADJUSTfCNT FOR A FALLIBLE COVARIATE
by
James Edwin McLean
August, 1974
Chairman: William B. Ware
Major Department: Foundations of Education
The purpose of this study was to determine if analysis of covariance
or analysis of covariance with Porter's true covariate substitution ad-
justment or neither is appropriate for analyzing pretest-posttest educa-
tional experiments with less than perfectly reliable measures.
Monte Carlo techniques were employed to generate two thousand sam-
ples for each of forty-eight sets of conditions. These conditions in-
cluded six combinations of reliability, two levels of sample size, two
levels pf gain, and the equality or inequality of pretest means. Each
sample was analyzed by both of the tested methods. The proportion of re-
jections for each method of analysis and the number of times the £
statistic for standard analysis of covariance exceeded that for analysis
of covariance with Porter's adjustment were recorded.
viii
The hypothesis, "The sampling distributions of the test statistics
from analysis of covariance with and without Porter's adjustments are
the same," was rejected at the .01 level of significance using the sign
test for each of the forty-eight sets of conditions.
To gain further insight into how the two methods of analysis dif-
fered, four factorial experiments were conducted using either the com-
puter generated alphas or powers as the criterion variables. The first
factorial experiment examined analysis of covariance when the mean gain
was zero in both groups, thus, the computer generated alphas were used
as criterion variables. The second experiment examined analysis of co-
variance with Porter's adjustment when the mean gain was zero in each
group. Again, computer generated alphas were used as criterion variables,
The third and fourth factorial experiments examined standard analysis of
covariance and analysis of covariance with Porter's adjustment when the
mean gain in only one group was positive. The four factorial experiments
included three factors. These were reliability at six levels, sample
size at two levels, and the equality of pretest means at two levels
(pretest means equal and pretest means not equal). In each experiment,
it was found that unequal pretest means combined with low reliability
were significant sources of variation.
Additional study showed that unequal pretest (covariate) means com-
bined with low reliability produced misleading results. More specifi-
cally, when either method of analysis produced a significant £ statis-
tic and there was a mean gain of zero in both groups but the pretest
mean of one group was less than that of the other, the adjusted post-
test means indicated that the group with the higher pretest mean had
the larger gain. Likewise, when there' was a positive mean gain in the
IX
group with the lower pretest mean, the adjusted posttest means indicated
the group with a mean gain of zero had the larger gain.
The results of this study combined with the results of others
concerned with analysis of covariance where the covariates are fallible
point to the inadequacy of the technique as it now exists. These
results should be kept in mind if the technique is to be used.
CHAPTER I
INTRODUCTION
. Regardless of the theory or hypothesis being investigated, educa-
tional researchers must, at some point, relate it to learning. One
common element of most definitions of learning is that a change must
take place in the learner (DeCecco, 1958, p. 243; Hilgard, 1956, p. 3;
Hill, 1971, p. 1-2). Thus most investigations in education are concerned
with identifying, measuring, and comparing these changes.
In the past decade, analysis of covariance has become a standard
procedure for comparing groups with different levels of ability. The
basic derivation of this procedure requires the assumption that the co-
variable does not contain errors of measurement (Cochran, 1957). In
recent years, several researchers have recognized that this assumption
is violated when. the qovari'ate is a mental test score (Lord, 1960;
Porter, 1967; Campbell and Erlebacher, 1970). Lord (I960) and Porter
(1967) have proposed adjustments to the analysis of covariance procedure
to use when the covariable is fallible, that is, when measurement error
is present in the covariable. The focus of this study is to determine
if analysis of covariance and analysis of covariance with Porter's
adjustment are different for a fallible covariate and if either is
appropriate under varying conditions of reliability, sample size, gain,
and pretest mean.
The Problem of Comparing Groups of Differing Abilities
The problem of comparing groups based on change manifests itself
prominently when the variables are mental measurements. Most physical
science measurements are made with the aid of some type of physical
instrument. If an engineer is interested in the weight gain of a metal
before and after a galvanizing process, instruments are available to
measure the weight of the metal to within at least one microgram. On
the other hand, if an educator is interested in the change in a student's
I.Q. before and after taking part in an experimental program, an error
of 5 points or more would not be uncommon. Based on the Wechsler
Intelligence Scale for Children with a standard error of measurement of
5.0 (Cronbach, 1970, p. 222), an error of 5.0 I.Q. points or more would
occur in approximately 32 percent of the measurements, assuming a normal
distribution. If the actual change in I.Q. were near zero for an
individual, it is srery likely that the error of measurement would exceed
the change itself.
The problem is compounded further as the groups being compared often
are not of the same ability level , Many of the recent programs featuring
innovative teaching practices for compensatory education are available
only for the most need^^ and the comparison group is then sampled from
the general population of students (Campbell and Erlebacher, 1970).
Programs such as Head Start and Follow Through have probably been the
victim of tragically misleading analyses, such as in the Westinghouse/
Ohio University study (Campbell and Erlebacher, 1970). Personal contact
with the evaluation of Follow Through projects has emphasized the
magnitude of the problem. Warnings about the inappropriate uses of
existing modes of analysis have come from several sources (Lord, 1960,
1967, 1969; Campbell and Erlebacher, 1970).
Remedies have been offered (Lord, 1960; Porter, 1967), but at this
time there is no conclusive evidence to indicate these remedies are
appropriate. Studies concerning the robustness of the analyses to
violated assumptions have also been in conflict (Peckham, 1970).
Methods of Comparing Groups of Differing Abilities
Jvio general approaches are available to compare groups with dif-
fering abilities. The first is to compare the average change in one
group with the average change in the other group by means of a _t test or
analysis of variance. The second is to use analysis of covariance, where
the pretest score is the covariate.
There are at least three methods of computing change scores to be
considered if the first approach is used. One method is to obtain raw
difference scores by simply subtracting each pretest score from its
corresponding posttest score. Another method is Lord's true gain (Lord,
1956) which was further developed by McNemar (1958). Basically, this
method requires the use of a regression equation to estimate the "true"
gain or change from pretest to posttest. A third method of measuring
gain, the method of residual gain , was proposed by Manning and DuBois
(1962). This method involves regressing the posttest scores on the
pretest scores and obtaining a predicted posttest score for each subject.
The indicator of change is the observed posttest score minus the pre-
dicted posttest score. This method is mathematically equivalent to the
standard analysis of covariance (Neel , 1970, p. 30-31), the second
approach.
The second general approach is the standard analysis of covariance
procedure found in many statistical texts (e.g.. Kirk, 1968; Winer, 1971).
One of the reported functions of a covariate is to adjust the treatment
means of the dependent variable for differences in the values of corres-
ponding independent variables (Hicks, 1965). Logically, it seems that by
using the pretest scores of groups differing in ability as a covariate,
the treatment means can be statistically adjusted for the differences in
pretest means, A deficiency of this method is the fallibility of the
covariate. A fallible variable is one which is not measured without
error or with perfect reliability (Lord, 1960). Lord (1960), recognizing
this deficiency in the use of analysis of covariance, derived an adjust-
ment to compensate for the fact that the pretest scores v^ere fallible.
Porter (1967) modified Lord's procedure for more than two groups and
empirically investigated the sampling distribution of Lord's statistic.
Porter's solution has been suggested as an alternative to the standard *-•'
analysis of covariance when the covariate is fallible.
Limitations Used in this Study
This study was designed to examine the characteristics of the two
covariance methods within the framework of simulated situations which
may occur in the analysis of a compensatory education project. Realis-
tically, several limitations were imposed. Even with these limitations,
the two methods of analysis were investigated under forty-eight combi-
nations of differing reliability, sample size, mean gain, and pretest
means.
The study was limited to two groups, henceforth called the gain and
no-gain groups. Investigating more than two groups would not only in-
crease the number of possibilities appreciably but would not be in line
with the objective stated previously, that is, comparing a compensatory
education group with a "comparison" group. The levels of the reliabil-
ities being investigated are also limited to the range .50 to .90. Sel-
dom do mental measurement instruments have reliabilities above .90 and
instruments with reliabilities below .50 are generally considered in-
adequate.
The levels of sample size were 10 and 100 subjects per group.
These quantities are representative of small and large sample sizes en-
countered in educational experiments.
Furthermore, the true score and true gain variances will be fixed
a priori with the true gain variance always four percent of the true
score variance. These figures are empirically based on Follow Through
achievement data in keeping with the aforementioned objective.
Procedures
The study compared the two methods of analysis under forty-eight
sets of conditions. The two methods were analysis of covariance and
analysis of covariance with Porter's adjustment for a fallible covariate.
Three levels of reliability and two levels of sample size were used.
The three levels of reliability will include situations where the re-
liability of the pretests are assumed equal and situations where they
differ to more closely simulate compensatory education project evalua-
tion. Each of these combinations of conditions was repeated for four
separate cases. Case I is where there is no-gain in either group and '^
both groups have equal pretest means. Case II is where there is gain in
only one group (the gain group) and both groups have equal pretest means.
Cases III and IV are with and without gain where the pretest means are
different.
For each of the forty-eight sets of conditions, two thousand samples
were computer generated and analyzed using both methods of analysis and
a .05 level of significance. Two thousand samples provide empirical
estimates of the fraction of type I and type II errors to within .01 of
their true values with ninety-five percent confidence. The samples were
generated from an assumed normal population. The generation technique
is one proposed by Box and Muller (1958) and modified by Marsaglia and
Bray (1964). This procedure generates random normal deviates with a
mean and variance of 0 and 1 respectively. These normal variables are
then transformed to attain the specified means and variances. The IBM
370/165 computer of the North Florida Regional Data Center was used for
the generation and analyses.
Using these analyses, two research questions were examined:
1. Is there any difference between the sampling distributions
of the test statistics of standard analysis of covariance
and analysis of covariance with Porter's adjustment?
2. What factors affect each type of analysis?
The first research question can be restated in terms of a null
hypothesis for each set of conditions:
The sampling distributions of the test statistics from
analysis of covariance with and without Porter's adjustment
are the same for each set of conditions.
This hypothesis was tested for each reliability, sample size, gain, and
pretest mean combination.
The second research question was examined in four factorial experi-
ments. Each factorial experiment included the factors of reliability at
six levels, sample size at two levels, and equality of pretest means at
two levels^ The first experiment consisted of examining standard analy-
sis of covariance with computer generated alpha values as the criterion
variables. The second examined analysis of covariance with Porter's cor-
rection also using computer generated alpha values as the criterion var-
iables. The third and fourth factorial experiments examined analysis of
covariance with and without Porter's adjustment respectively, using
computer generated powers as criterion variables.
Relevance of the Study
This study is designed to compare two recommended methods of ana-
lyzing educational experiments under known simulated conditions which
are presumably realistic. The results should indicate whether either
of the methods is appropriate for its recommended use, thus giving edu-
cational researchers some direction when faced with the choice. If both
methods were shown to be appropriate, the study could determine which
one is superior. The study should also indicate if either method of
analysis is appropriate for a set of restricted conditions, e.g. for
only certain levels of reliability.
Organization of the Dissertation
A statement of the problem, a description of possible solutions,
and an overview of the procedure of this study has been included in
Chapter I. A comprehensive review of the related literature is provided
in Chapter II. This review includes a historical overview of the pro-
blem and a detailed description of the methods being compared. A
description of the procedures followed for this study is contained in
Chapter III. The data and analyses are presented in Chapter IV and a
discussion of the results is provided in Chapter V. A summary of the
study is provided in Chapter VI.
CHAPTER II
REVIEW OF THE RELATED LITERATURE
Comparing groups of differing abilities has been a problem for some
time. The literature has included discussions of possible solutions for
at least the past three decades. Many of these discussions have been in
conflict with one another, and there is still no general agreement. The
evaluation of federal compensatory education projects in the last decade
has intensified the debate and the need for a theoretically sound and
practically useful solution. Some of the solutions proposed over the
years are matching subjects, gain scores, and analysis of covariance.
A discussion of these solutions and why they may not be considered
satisfactory is provided in this chapter. The major properties of
analysis of covariance are also considered. Because Porter's adjustment
for a fallible covariable in analysis of covariance is not readily avail-
able in the literature, its derivation is provided in this chapter.
Historical Review of the Problem
A paper by Thorndike (1942) examined in detail the fallacies of
comparing groups of differing abilities by matching subjects. The crux
of his argument was that the regression effects were systematically dif-
ferent "whenever matched groups are drawn from populations which differ
with regard to the characteristics being studied," (p. 85).
During the late 1950 's and early 1960's the literature was inundated
with papers on how to or how not to measure gain or change. One of the
leaders during this period was Lord (1956, 1958, 1959, 1963). His
proposal for estimating the "true" gain was originally set forth in 1956
with further developments in 1958 and 1959. McNemar (1958) extended his
results for the case of unequal variances among the groups. Garside
(1956) also proposed a method for estimating gain scores and Manning and
DuBois (1962) presented their derivation of residual .gain scores.
The debate over what techniques, if any, should be used for measuring
gain or change was at its peak in the early 1970's. Cronbach and Furby
(1970) concluded that one should generally rephrase his questions about
gain in other ways. Marks and Martin (1973) underscored the Cronbach
and Furby (1970) conclusion. O'Connor (197.2) reviewed developments of
gain scores in terms of classical test theory. Neel (1970) employed
Monte Carlo techniques to compare four identified methods for measuring
gain. The compared methods were raw difference. Lord's true gain,
residual gain, and analysis of covariance. Under equivalent conditions,
he found that Lord's true gain tended to produce a greater significance
level than the user would intend, that is, a higher fraction of type I
errors than alpha.
The analysis of covariance method has been widely recommended by a
number of authorities in the field (Thorndike, 1942; Campbell and Stanley,
1963; O'Connor, 1972). Campbell and Erlebacher (1970) point out that the
Westinghouse/Ohio University Study was evaluated, possibly incorrectly,
using the analysis of covariance. Lord (1960, 1967, 1969) has sounded
a warning about its use that has been echoed by others (Werts and Linn,
1970; Campbell and Erlebacher, 1970; Winer, 1971). The warning stressed
that the analysis of covariance requires the assumption that the covariate ''''^
is measured without error. Lord (1960) has proposed an adjustment which
10
was later generalized by Porter (1967). The adjustment is based on the
substitution of the true score estimate for the observed value of the
covariate.
Analysis of Co variance
The analysis of covariance procedure was originally introduced by
Sir Ronald A. Fisher (1932, 1935). According to Fisher (1946, p. 281),
the analysis of covariance "combines the advantages and reconciles the
requirements of two widely applicable procedures known as regression and ^
analysis of variance." The procedure is well documented in contemporary
texts (Snedecor and Cochran, 1967, p. 419-446; Kirk, 1968, p. 455-489; :
Winer, 1971, p. 752-812). Analysis of covariance is a popular technique
in both the physical and social sciences.
Among the principal uses of analysis of covariance pointed out by
Cochran (1957), p. 264) is "to remove the effects of disturbing variables
in observational studies." It was thought that by using the pretest score
as the covariate and comparing two groups with analysis of covariance,
the effects of different pretest scores could be eliminated. Lord (1960)
pointed out that this was not necessarily the case when the covariate
was fallible. The assumptions necessary for the analysis of covariance
are the same as those for analysis of variance with the addition of the
following:
/
1. The covariates are measured without error. /
2. The regression coefficient is constant across all treatment
groups (Peckham, 1970).
Violation of the first assumption induces a bias in the analysis
of covariance because of "the presence of 'error' and 'uniqueness' in
11
the covari ate, i .e. variance not shared, by the dependent variable. If
the proportion of such variance can be correctly estimated, it can be
corrected for," (Campbell and Erlebacher, 1970, p. 199). This position
was upheld by Glass et al . (1972). The basic algebraic derivation of
this correction was presented by Lord (1960) and expanded by Porter (1967]
Analysis of Covariance with Porter's Adjustment for a Fallible Covariate
Lord's (1960) derivation of the analysis of covariance adjustment
was limited to two treatment groups and is slightly more difficult than
is Porter's (1967) procedure. Thus, Porter's procedure will be derived
here and used in the analysis. Porter's adjustment is based on the sub-
stitution of the estimated true value for the covariate.
Let X denote a fallible variable (e.g. a pretest score) , r an esti-
mate of the reliability of X, and T-j the true value of the variable X^- .
Let T-j denote the estimated true score of X-j . The definition of T-j for
each individual is
(1) Ti = X + r(Xi-X),
or T^. = rX^- + X(l-r),
Porter (1967) derived the mean and variance of T-j as follows. By
definition, the mean of all T-j's is:
(2) T= ^Ti^
IT'
= E[rX^+)r(l-r)]
ii
by substitution. Thus
T = rX + NX-NrX ,
N
= X ,
12
where N denotes the sample size and it is understood that i is summed
over the values 1, 2, ---, N.
Also, by definition, the variance of t is
(3) S^ = E(fi-X)^ ^
N-1
= r2s2
X'
and
(4) Sj^ = z(fi-)()(Yi-Y) ,
N-1
where Y is the dependent variable.
Based on these results. Porter (1967) derived an analysis of covar-
iance procedure replacing the fallible covariate, Xj , with the estimated
true score, T^-. The analysis of covariance requires the computation of
analysis of variance sums of squares for the dependent variable, the co-
variable, and on the cross-products of the dependent variable and the co-
variable. Porter (1967) showed that the use of estimated true scores for
the covariable did not affect the analysis of variance of the dependent
variable, Y. The changes found by Porter (1967) in the other two cases
are as follows:
For the analysis of variance of the estimated true scores, f,
(5) SS^^ = lE(fij-f.j)2
= r2j:z(Xij-X.j)^,
13
where SSj,, denotes the within groups sum of squares.
(6)
SSn. = nE(t.-i-f..)'
= nE(X..-X..)^
where SSg^ denotes the between groups sum of squares,
(7)
SSy. = ZE(Ti-j-T..)'
T
= r2zzX?.+(l-r)E
n N
where SSj^ denotes the total sum of squares, n denotes the number of
(T,Y) pairs per treatment group, and N denotes the total number of (t,Y)
pairs. In a similar manner, the cross-products sums of squares are:
(8)
(9)
(10)
SSwfv = ^^^(Xij-X.j) (Yij-Y.j),
SSBfY ^ "^(X.j-X..)(Y.j-Y..)^
SSt. = rEZX,-iY,--i+(l-r)E
(i:Xij)(zY,-j)
— J^
Thus, the adjusted sums of squares are
(rSS )2
SSu' = SSi^ ^XY ,
Y ~2^7 "
(11)
SS,,
X
^XY
SS,,
(rSSw + SSn )2
(12) SS't = SSt - XY XY
^^^\ \ SSb^
(13)
SS ' = SS' - SS '
14
Note that the adjusted v/ithin groups sum of squares remains un-
changed by the substitution of T for X but the substitution does alter
the adjusted total sum of squai'es and consequently, the adjusted between
groups sum of squares.
A question arises concerning what value to use as the estimate of
reliability in the formulae. A solution to this problem was proposed
by Campbell and Erlebacher (1970).
In a pretest-posttest situation one may find it reasonable
to make two assumptions that would generate appropriate common-
factor coefficients. First, if one has only the pretest-posttest
correlations, one may assume that the correlation in the experi-
mental group was unaffected by the treatment. (We need a survey
of experience in true experiments to check on this.) Second, one
may assume that the common-factor coefficient is the same for
both pretest and posttest. Under these assumptions, the pretest-
posttest correlation coefficient itself becomes the relevant
common-factor coefficient for the pretest or covariate, the
"reliability" to be used in Lord's and Porter's formulas, (p. 200)
This recommendation will be followed in this study, thus the corre-
lation between the pretest scores and posttest scores will be used as
the reliability estimate in, the formulae.
S umma ry
As noted, analysis of covariance and analysis of covariance with
Porter's adjustment have been recommended by several authors. Campbell
and Erlebacher (1970) computer generated data for two overlapping
groups with no true treatment effect and concluded that the analysis
of covariance method was inappropriate and that analysis of covariance
with Porter's adjustment should only be undertaken with great tenta-
tiveness. Porter (1967) computer generated data to compare the _F
sampling distribution with the theoretical £ distribution using his
adjustment. He concluded that samples of 20 or larger were needed to
have a useful approximation to the theoretical F distribution. He also
15
found that the estimation degenerated further when the reliability was
less than .7. No study was found which compared both techniques under
similar conditions for both gain and no-gain groups.
CHAPTER III
PROCEDURES
The analysis of covariance and the analysis of covariance with
Porter's adjustment were compared under forty-eight sets of conditions
on the basis of computer generated data. Six combinations of reliability,
two sample sizes, two levels of gain, and two different sets of pretest
means were used. Both equal and unequal reliabilities were used in
the comparisons. A random sample of two thousand observations was gen-
erated under each combination of conditions and analyzed by analysis of
covariance with and without Porter's adjustment.
The sampling distributions of the two analyses were statistically
compared with a sign test for each of the forth-eight sets of conditions.
The computer generated alpha values and powers were then used as the
criterion variables in four factorial experiments. Subsequent a poste-
riori analyses were performed where warranted.
The Model
A standard model was used to represent the pretest and posttest
scores of one subject. The model follows the traditional measurement
approach as found in Gulliksen (1950) or Lord and Novick (1968) and
extended to gain score theory. by O'Connor (1972).
(14) X = T + E^
and
(15) Y = T + G + E2
16
17
where .
X = observed pretest score,
Y = observed posttest score,
T = true pretest score
G = true gain,
£■]= random measurement error in pretest score,
£2= random measurement error in posttest score.
The following properties about E-j and E2 are assumed to exist:
The errors E^ and Eo
i) have zero means in the group tested,
ii) have the same variances for both groups,
iii) are independent of each other and of the true parts of
each test.
It is further assumed that T and G are independent (across subjects) and
that all components follow a normal probability distribution. These
assumptions parallel Lord's' (1956) stated and implied assumptions.
Selecting the Reliabilities
Reliability is related by definition to the variances of the ob-
served scores, the true scores, and error. This section shows how that
relationship can be used to establish desired reliabilities. The basic
definition of reliability (Helmstadter, 1964, p. 62) is given by
equation (16) where the symbol, p^^, denotes reliability.
(16) p^x = 4~4
2 ■
°X .
Then,
;i7)
a2 = o2(l_p )
■1
XX
Choosing the variance of X a priori to be 100, the variance of E-\ can be
found in the following manner as the reliability of X assumes different
values:
(18)
-1
100(1 -p )
XX
The independence of T and E-] in (14) implies
(19) "2
^x = ^T ' i.
Combining (18) and (19) and solving for a^ yields
(20)
1 = ^°Sx-
The posttest variances can be selected in a similar manner. Based
on the assumptions.
(21)
a^ + a
? + a2
The variance of the gain scores is chosen in the manner prescribed in
Chapter I. Then, combining (16) and (21),
;22l
4.
+ 2
'T
P|YY
where pyy denotes the established reliability of the posttest scores.
Thus it can be seen that the effect of selecting specified reliabilities
can be obtained by selecting the variances of E-] , E2, and T in accor-
dance with (18), (20)
, and (22) respectively.
The Regression of Y on X
Recall that one of the assumptions necessary for analysis of co-
variance (Cochran, 1957) is that the regression slopes of the dependent
19
variable on the covariate must be equal for each treatment group. Let
gy-x denote the regression slope for one treatment group.
Then,
(Ferguson, 1971, p. 113),
(23)
Py-X = PXY • ^ '
^X
implies
(24).
ev.Y = "XY
2^2
"T "" °E^
The covarianee between X and Y is equal to the variance of T since it
has been assumed that all the components of the pretest and posttest are
independent except T rwith itself. Therefore
2
(25) By-x = ^T ,
2 2
and
(26) gy.x = Pxx
by equation (16). Thus, the slope, Y on X, of any group is equal to the
reliability of its pretest. If the reliabilities of the pretests were
the same for both groups, the assumptions concerning slopes would be
satisfied.
From equation (16), it can be seen that the reliability of a test
is dependent to a certain extent upon the variability of the sample for
which it is given. In equation (16), o^ is in both the numerator and
A
denominator of the fraction with a^ being subtracted from the numerator.
2
As 0^ increases both the numerator and denominator increase by the same
2
amount assuming o^ is not changed thus, pxx increases. Two groups of
equal ability would likely produce similar variances, hence similar
reliabilities. However, the technique is often used for comparing
20
compensatory programs with a comparison group sampled from the general
population of untreated children in the same community such as with the
Westinghouse/Ohio University study (Campbell and Erlebacher, 1970). These
comparison children tend to be higher in ability than the treatment group.
The resulting differences in variation tend to produce different reli-
abilities. In order to simulate such a situation this study used dif-
ferent reliabilities for the gain and the no-gain groups in addition to
the comparisons when the reliabilities are equal. The reliabilities for
the gain group are decreased by fifteen percent which roughly approxi-
mates the reduced variation empirically observed in Project Follow
Through achievement data.
Selecting Heans
The true score mean was set a priori at 100 when both the gain and
no-gain groups have equal means. The situation in which the gain group
has a lov;er mean was also analyzed. In this case, the true score mean
of the gain group was set at 80. These values have been empirically
chosen based on Project Follow Through data. Based on the assumptions,
(27) E(X) = E(T+Et) = y^,
hence the mean of the observed pretest scores is equal to the mean of
the true scores. Also
(28) E(Y) = E(T+G+E2) = y-^+yg.
Thus, the mean of the observed posttest scores is equal to the sum of
the means of the true scores and the gain scores.
Clearly in the case of the no-gain group and in both groups where
no gain was used, the mean gain was zero. The selected value of yQ for
the gain situation was based on power considerations, that is, y^ was
21
chosen such that the power of an £ test for analysis of covariance was
.50.
A linear model representation of the analysis of covariance for two
groups is
(29) Y = eg +BiX+e2W+e
where X is the pretest score (covari ate), Y is the posttest score, and
W is a dummy variable designating group membership (W=l if gain group,
0 if no-gain group). Testing the hypothesis that 62 ""s equal to 0 in
equation (29) is equivalent to the F test for treatments in the analysis
of covariance procedure. It can be shown that B2 i" equation (29) is
equivalent to the mean gain, pg. The mean of the posttest for the gain
group is
(30) E(Yg) = 30+^1^X3+^2
where Yq is a posttest score for the gain group and p^ "is the mean pre-
G
test score for the gain group. Likewise, the mean of the posttest for
the no- gain group is
(31) E(Yng) = Bo+6iyXf^e
where Y|^q is a pretest score for the no-gain group, pv is the mean pre-
NG
test score for the no gain group. But it has been assumed that p^ and
PXfjQ are equal. Furthermore, the gain group has mean gain, pg and the
no-gain group has mean gain zero, thus
(32) pg = E(Yg)-E(YNg) = B2 '
Hence, choosing the value of 62 'that yields a power of .50 is equivalent
to choosing a value of pg to produce a power of .50 in the analysis of,
covariance procedure.
22
This power can be obtained from the following probability state-
ment:
(33) Pr[t*>tJ = .50
where t_* is a noncentral t_ statistic. This expression can be approxi'
mated by the substitution of a z, statistic for t*.
(34)
Pr
^2-H > t
= .50
Thus a value of 60 can be chosen such that the substituted z_
statistic is equal to t Q25 with the appropriate degrees of freedom.
This value of &2 ""s the value of the average gain, \ir, such that the
power of analysis of covariance is .50 under the condition of perfect
reliability. In order to find this value of yg, a numerical expression
for or is needed.
The variance of 63 > of » can be approximated in the following
manner. Karmel and Polasek (1970, p. 245) state that a? is
62
(35)
2 2
J- = ar;
32 Y
e(x-x)
v^2
[e(X-X)2][e(W-W)^]-[i:(X-X)(W-W)]^
Dividing both the numerator and denominator by N^ and substituti
ng
population variances for sample variances, a? is approximately equal
62 , . .
to the following:
(36)
32,
2 2
^Y °X
N 2 2/ .2
Ov o,, -{a^J
'X "W
XW'
The quantity, oy^^, the covariance between X and W has been assumed equal
to zero thus, in equation (36), the Ow's divide out. Hence
23
(37) a? = a?
2-
Under the conditions assumed for the model, the variance of Y, '^Y,
2
is equal to 4. The variance of W, °W can be computed to be .25. Thus,
2 2
by substitution, o- equals .80 when N is equal to 20 and ^g equals .08
1^2 2
when N is equal to 200.
Hence, for N = 20, Mq can be found by the following expression:
(38) yg = t ^^^ ,^a;- = 2.1] /rm = 1.88.
20 ~-U^o,l/ ^2
Likewise, for N = 200, yg can be found by solving equation (39).
(39) yg =t 025 igyaa = 1. 97/708 = .56.
200 -u^o^iy/ ^2
The approximated values of yg were tested using Monte Carlo generated
variables and found to indeed produce a power of .50.
The approximated values of yg under each set of conditions are
shov/n in Tables 1 and 2.
24
TABLE 1
CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN
THE RELIABILITIES FOR' BOTH GROUPS WERE EQUAL
GROUP
GAIN
GROUP
PRETEST
PRETEST
SAMPLE
MEAN
MEAN FOR
MEAN FOR
RELIABILITY
SIZE
GAIN
GAIN GROUP
NO- GAIN GROUP
.90
10
0
100
100
.90
10
0
80
100
.90
100
0
100
100
.90
100
0
80
100
.70
. 10
0
100
100
.70
10
0
80
100
.70
100
0
100
100
.70
100
0
80
100
.50
10
0
100
100
.50
10
0
80
100
.50
100
0
100
100
.50
100
0
80
100
.90
10
1.88
100
100
.90
10
88
80
100
.90
100
56
100
100
.90
100.
56
80
100
.70
10
88
100
TOO
.70
10
88
80
100
.70
100
56
100
100
.70
100
56
80
100
.50
10
88
100
100
.50
10
88
80
100
.50
100
56
100
100
.50
100
56
80
100
25
TABLE 2
CONDITIONS UNDER WHICH COMPARISONS WERE MADE WHEN
THE RELIABILITIES FOR BOTH GROUPS WERE UNEQUAL
RELIABILITY
RELIABILITY
GROUP
GAIN
GROUP
PRETEST
PRETEST
FOR GAIN
FOR NO- GAIN
SAMPLE
MEAN
MEAN FOR
MEAN FOR
GROUP
GROUP
SIZE
GAIN
GAIN GROUP
NO-GAIN GROUP
.76
.90
10
0
100
100
.76
.90
10
0
80
100
.76
.90
100
0
100
100
.76
.90
TOO
0
80
100
.60
.70
10
0
100
100
.60
.70
10
0
80
100
.60
.70
100
0
100
100
.60
.70
100
0
80
100
.42
.50
10
0
100
100
.42
.50
10
0
80
100
.42.
.50
100
0
100
100
.42
.50
100
0
80
100
.76
.90
10
1.88
100
100 '-
.76
.90
10
88
80
100
.76
.90
100
56
TOO
100
.76
.90
■ 100
56
80
100
.60
.70
10
88
100
TOO
.60
.70
10
88
80
TOO
.60
.70
100
56
100
100
.60 ~
.70
100
56
80
100
.42
.50
10
88
100
100
.42
.50
10
88
80
100
.42
.50
100
56
TOO
100
.42
.50
100
56
80
100
26
Generation of Random Normal Deviates with Specified Means and Variances
The study required the use of computer generated normally distrib-
uted random variables with specified means and variances. Two thousand
sets of variables were generated for each of the forty-eight sets of
conditions. Muller (1959) identified and compared six methods of gen-
erating normal deviates on the computer. A method described by Box and
Muller (1958) was judged most attractive from a mathematical standpoint.
According to Muller (1959, p. 379), "Mathematically this approach has the
attractive advantage that the transformation for going from uniform deviates
to normal deviates is exact." This method was endorsed by Marsaglia and
Bray (1964). They modified the algorithm to reduce central processing
computer time without altering its accuracy.
The method first requires the generation of two independent uniform
random variables, U^ and U2, over the interval (-1, 1). The variables
and
Z^ = U^[-2 ln(U^+U^) / {U^ + ul)2^^^
I2 = U2[-2 In(U^ui) / (U? + \ilW^^
will be two independent random variables from the same normal distrib-
ution with mean zero and unit variance. The variables were then trans-
formed to have the desired means and variances.
Analysis for Comparing the Selected Methods
Each of the two thousand sets of generated data for the forty-eight
sets of conditions was analyzed by analysis of covariance and analysis
of covariance with Porter's adjustment. The number of times the F
statistic from analysis of covariance exceeded the £ statistic from
analysis of covariance with Porter's adjustment was noted along with
the proportion of rejections by each method of analysis.
27
A sign test (Siegel, 1956, p. 63-67) was then performed for each
set of conditions to test the null hypotheses in Chapter I, that is,
"the sampling distributions of the test statistics from analysis of
covariance with and without Porter's adjustment will be the same for
each set of conditions." These tests were run at the .01 level of
significance.
The fraction of rejections noted (alphas and powers) was then used
as the dependent variable in four factorial experiments to gain further
insight into what factors affected each method of analysis being studied.
Each factorial experiment had three factors. These were reliability
which included six combinations of reliability used in the study, sample
size, which included n =10 and n = 100 subjects per group, and the
equality of pretest means, which included a level where both pretest means
v/ere equal and a level where they differed. The layout of the factorial
experiments is illustrated in Table 3. The factorial experiment illus-
trated is the situation for which there was no gain in either group and
the standard analysis of covariance was used. Thus, the dependent variables
are the alpha values generated by the computer. When significant main
effects or interactions occurred, the appropriate a posteriori analytical
procedures to locate the sources of the variation were followed.
28
LU O
<:
CD
o
O r-
O <
1— I in
LU _J
I— <;
eC LU
CO O
LU CC
Q tJ3
i-i cC
O
2: rvi
UJ cC
LU «a;
<c •-•
0; LU
ca UJ
I/) 00
Q- s:
Q. 2:
CHAPTER IV
RESULTS
The. number of times the £ statistic from the standard analysis of
covariance exceeded the F statistic from analysis of covariance with
Porter's adjustment is listed for each set of conditions in Tables 4,
5, 6, and 7. These quantities were used as test statistics for the sign
tests used. to test the hypothesis, "There is no difference between the
sampling distributions of the test statistics from analysis of covariance
with and without Porter's adjustment." This hypothesis was tested for
each of the forty-eight sets of conditions at the .01 level of signifi-
cance.
Siegel (1956) stated that a large sample test statistic for the
sign test is
(40) z = X-.5N
757R~
where N is equal to the number of pairs of observations, x is equal to
the number of times the first measurement of the pair exceeds the second
measurement of the pair, and z_ is the standard normal variate. For a
level of .01, the null hypotheses would be rejected when z was less than
-2.33 or greater than 2.33. This is equivalent to rejecting the null
hypotheses when x was less than 949 or greater than 1051 and N equals
2000. Thus, an inspection of the last column of Tables 4, 5, 6, and 7
reveals that the null hypothesis was rejected for each of the forty-
eight sets of combinations.
The analysis of variance summary tables avq presented in Tables 8,
29
30
9, 10, and 11. The analysis of variance summary table for the case when
the Monte Carlo generated alpha values from the standard analysis of
covariance were used as the criterion variables is presented in Table 8.
The analysis of variance summary table for the case when the Monte Carlo
generated alpha values from the analysis of covariance with Porter's ad-
justment were used as the criterion variables is presented in Table 9.
The analysis of variance summary tables for the cases when Monte Carlo
generated powers were used as the criterion variables for standard
analysis of covariance and analysis of covariance with Porter's adjust-
ment are presented in Tables 10 and 11, respectively. Each analysis of
variance table includes analyses of the simple effects where they are
warranted. Scheffe's S method for testing linear contrasts is included
in Table 8. These linear contrasts are defined in Table 12. Each F
statistic which exceeds the critical value at the .05 level is denoted
by an asterisk.
A result of particular interest is applicable to analysis of co-
variance both with and without Porter's adjustment. That is, when
a spuriously high fraction of significant £ statistics occurred when
there was a mean gain of zero in both groups and the pretest means
differed, the adjusted posttest means indicated that the gain was in
favor of the group having the largest pretest mean. This result was
more pronounced for lower reliabilities.
When the gain group had a positive gain, the no-gain group had a
mean gain of zero, and the no-gain group also had a larger pretest mean,
a similar situation occurred. In this situation, when a significant £
statistic occurred, the adjusted posttest means usually indicated the
31
no-gain group had recorded the larger gain. Again, these results be-
came more pronounced as the reliability of the scores were reduced.
32
CO Q Q (yO
LU Q; LU
: 2: <: Q ct:
►-I O LU LU
I— z: uj t—
cC o d:
I u. I— X o
o t/) LU Q.
1— —I
- ,0 1— (/5
<c
ll| h-rD
UJ ID
00 3 '^
ca 0-
Ll. I— •— 1 Q
UJ LU
0 -Z. 00 LU cC
nr
<: :^ 0
3 LU
2: 0 _i 2: 00
C£.
01— eC cc-
OJjJ
1— 1 u_ s: 1— ' cc:
1— i-H <: d; LU
0 z: cc (-
Q 1/5
=c 0 q: > q:
LU LU
q: K-. 0 0 0
Q 1-1
Ll_ t/1 U. <_3 Q_
LU 1—
1
LU 1—1
0 —1
X 1—1
(O
LU m
1 00 LU
■=c
2: ►-I 0
2:
Q cC
2:
t/0 O
Lu |0
u. (— <; 00 2:
o 2: 1— >- <
=a: 00 _i •-<
^ o ■ <c ct;
o i-< cc: z: <c
I— I Li_ o <:- >
I— i-H u. o
02: Q O
=1 CD 00 Q;
ae:^-<- <: L:.
Ll- 00 u-Iq O
t/> O H-i
UJ cC Q.
I— :z o ZD
LU <: I o
a: LU o q:
Q- 2: ^ CD
I— z z: rs
LU CC t-l o
QC LU -CC Di
D- 2: O CS
00
00
CO
00 :z
Q. LU m rD
s: rvj 0 0
LU =t
cc •-< < d;
0 CD
OO CO LU CD
0 <c
1— ^ d;
<C 3
H-. K-. 0
cc:
_J ccC 0
U-
•-• 0 00
OQ 1
cC 0 o.
»-< 2: rD
_l 0
LU Li_ a:
q: 0 CIS
;z 00
en
t— H
<c <: D-i
l~H
ts r)
, 1
0
UJ
u_ cc:
q:
0 Ci3
-C
+J
Q. C
r—
(O
rtS
CJ
•r—
-c*-
QJ
•r-
>
C
(J)
S-
•1—
fl)
t/l
T3
Wl
0
•"-
S-+->
ro
t/)
33
CO Q Q t/)
ljj q: uj -
q: 2: cc Q o;
UJ 1— I Q UJ LU
CQ I— :s LU I—
S < O Di
ZD U- (— X O
^ o 1/1 UJ o.
UJ _l
2:
zd <:
to Li. 31 I—
h- =>
- ,0 1— to
0-
Lu ^ =1
UJ UJ
00 3 --0
Qi
u_
h- ^-1 Q
UJ 1—
c
2: to UJ =3;
zc 0
< :>- <J
3 ZZ
z 0 _i 2: 1/1
0
H-l < <-
riiii
U_ Di
h-
u- sr 1-1 Qi
1-. <r q: UJ
UJ
0 z: <c 1—
Q 3
ec CD q; > ct:
UJ
cc:
•-< 0 0 0
Q to
U-
t>0 U. t_) O-
UJ UJ
UJ 1—1
0 (-
X h-<
fO
UJ _l
1 to UJ
z: >-< 0
LJ-1^
u.
1— <C to ;z
0 2: H- :^ <c
h—i
er CO _l *-•
CO —1
2: 0 <: Di
UJ UJ
c
t-i K s: cc
2: ct:
Li. 0 =C >
h-
l-l u_ 0
1— J—
0 ^^ Q 0
t/o
ca: C3 to cc
U_ UJ
cc:
i-< - ,ca: u_
0 I—
u_
cou.|Q 0
UJ
D£. CC
UJ Q-
CQ
H-Li_ 2:
S: UJ
to 0 ►-1
ZD m
UJ CC D-
z: 1—
uj< 10
Q Q
CCUA 0 CC
:z 2:
n.2: ^ es
cc ca:
CO a.
- ,zo
H-Li-
Ui_ 0
00 0
q:
UJ CL
1— 0
1— ^ Z ID
■z.
UJ cC •— 1 0
<c ct:
q; UJ ca; q:
C_) UJ
o-s: ts CD
Q. UJ 3: r3
s: rvi o o
ca: M cj; d;
t/l to UJ ts
eC CO
I-. KH 0
CC =3:
_J =3: C_3
U. 3
>-' CD to
CO r
< 0 Q-
i-i :r rD
—1 0
UJ u_ or:
a; 0 CD
J o
UJ u_ o;
q: o CD
CM O
^ O
o o
Q- C
r— nj
n3 tj
CU in
S-4J
"3 t/>
t_> <u
■•-> C
c cn
rO J3
34
t/) a Q 1/1
LU O^ UJ
QC ^ <c Q cc:
UJ I— I Q LU UJ
CQ I— 2: LU I—
2: eC o q:
ZD Li_ 1— X CD
'Z O t/1 LlJ Q.
a:
00 3: --D
f- _i
LJ_ 1—
1-1 c:
<i:
0 :z:
00 UJ cr
LU =3
ct >- 0
cc cy
2: 0
_J ^ I/:
UJ LlJ
0 <-H
< <c -
3:
1—1 U.
:z 1-1 Di
3 LU
1— •-<
< a: LU
Qi
0 IZ
<: H-
;^
< CD
o; > q:
q: ►-<
00c
LU (/I
Lu 0 a.
Q t/1
LU LU
Q I-.
(T3
LU I—
1 00 LU
LU 1— '
2: HH C_J
0 _J
Ll_ 1— ea: U1 2:
X 1— 1
0 2: I— ::^ <C
LU CQ
■=c
00 _l l-H
<:
s 0
■< Qi
unn
0 <-<
q: 2: <c
1—1 u_
0 =a: >
LU
H- '-'
u_ 0
4/1 a:
0 :z
Q 0
LU
<: 0 oo d:
s: 1—
cc: 1-1
- ,■< U-
i-H 00
Ll-l/O
u-|q 0
h- UJ
t-
U_ LU
f-
LL. -z.
10 0 d;
00
0 i-<
D_
UJ
<; Q-
LU Qi
h-
■Zi^-Zi
_J LU UJ
LU
<C 1 c
CQ CQ.IC
DC
ui 0 a;
■< 2: 1—
Q-
s: 2: 0
1— =3
^0
2:
H-
u_
Q =C
00
0
2:
LU
Q.
<: Q-
1—
2: ^ =)
^
LU
CC H-l 0
</i 0
CC
ui =a: q:
- i'^
a.
2: CD crs
U-lcD
(— :z
U-
Z M
0
■^ <C
D-
0 0
~j^
2: 2: =)
1— 1
cX,
i-i 1-1 0
Ll_ LU
LU =1 ct Q;
t-i n:
S: 0 CD CD
2: h-
0
1-1 2:
u.
c/o •— 1
LU
0
_J
Q-
LU 2:
D.
Lu n: ZD
0 1^
s: rvi 0 0
<:
■<
1-1 < q:
2: 0
00
I/) LU CD
0
!-• in
H- <c
(/I
I— 2: Qi
H-i 1-1 O
— J CC O
1-1 CD OO
CQ I
<: o CI.
^ to CvJ
o <— O
I— r— 00
35
U_| LL.\
00 Q Q (•)
LU Di LJ -
q: s: ct Q cc:
LU >— I Q LU LU
-CQ I— 2: LU f—
2: ca: o q;
ZD U- I— X O
2: O CyO UJ CL.
- ,0 h- C/)
c: _j
u_| h-<,:d
LU eC
tr> iS'-Z)
x: rD
Li_ 1— 1— 1 Q
1- 0-
0 2: 00 LU <:
UlI
<: >- 0
LU
S 0 _J Z 00
oc h-
0 i-H <c ■<-
LU 0
1— 1 Li_ ^ H-i q;
x: ^
1— t— 1 cc ce: LU
3
0 z: =3: 1—
LU
ei 0 Qi > q;
U-|LU
q: i-H 0 0 0
Ll_ 00 LU 0 Q.
^
to
Q CO
fO
LU LU
1 GO LU
1 1 1 1— i'
^ I-. 0
0 1—
u- 1— <c CO z:
X 1—1
0 2: H- >- cC
LU _1
ca: 00 _i HH
^ cj cc q;
t/>,CQ
0 M Di 2: <c
Ll-K
i-H u- 0 =1: >
1— M u. 0
00 _i
02: Q tJ
LU LU
=1 c3 GO q;
S a:
q; K-.- <c Li-
1 — 1
Ll. 00 U_|Q 0
1— (—
GO
U- LU
1— u_ ^
0 f-
oo 0 1— 1
UJ
LU ec a.
cd <x.
H- sr trs rj
LU Q-
LU <: 1 0
CQ
c; LU 0 d:
2: LU
Q- 2: z: C3
n 3:
S t—
1— U-
Q Q
CO 0
2: ^
UJ Q.
<C <C
(— 2: 2: ZD
LU ■< 1— 1 0
OO Q-
a: LU cc d;
- rD
Q- 2. es CD
^le .
CC HH KH O
LU <C <£ Qi
S CU CD C5
U- >-H
a.
LU n: Z3
0
2: rvj 0 0
Z
=1:
l-H < Di
^ HH
00
00 LU CD
0 CC
l-H 0
H-
00
CJ 00
>- LU
S^
1— 2: Di
l-H 1-1 0
Ll_
_J cC 0
>-l CD 00
CQ 1—1
cc <: Q-
h-i CD :3
VO 1—
r— o
00 cr^
LD
0
10
CM
0
CO
CO
CO
0
CXJ
1 —
0
0
' —
^
36
TABLE 8
ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS FROM THE
STANDARD ANALYSIS OF COVARIANCE AS THE CRITERION VARIABLES
SUM OF
DEGREES OF
MEAN
SOURCE
SQUARES
FREEDOM
SQUARE
F
Pretest Mean
1.59960
1
1.59960
(544.08)
PM at R-,
.07049
1
.07049
23.98*
PM at R2
.33582
1
.33582
114.22*
PM at R3
.40513
1
.40513
137.80*
PM at R4
.16687
1
.16687
56.76*
PM at Rn
.33466
1
.33466
113.83*
PM at Rg
.42510
1
.42510
144.59*
PM at SSio
.11525
1
.11525
39 . 20*
PM at SS^QQ
2.10000
1
2.10000
714.29*
Reliability
.10996
5
.02199
(7.48)
R at PMi
.00008
5
.00001
<1.00
R at PM2
.22278
5
.04455
15.15*
n
.00018
1
.00018
<1.00
4-2 ■
.10306
1
.10306
35.05*
"^'3
.73933
1
.73933
251.47*
*4
.04743
1
.04743
16.13*
Sample Size ,
.59977
1
.59977
(204.00)
SS at PM-,
SS at PM2
.00005
1
.00005
<1.00
1.21540
1
1.21540
413.40*
PM X R
.11291
5
.02258'
7.68*
PM X SS
.61568
1
.61568
209.41*
R X SS
.01470
5
.00294
1.00
Residual
.01468
5
.00294
Total
3.06730
23
*Sample statistic greater than critical value at .05 level
37
TABLE 9
ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED ALPHAS
FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT
AS THE CRITERION VARIABLES
SOURCE
SUM OF
SQUARES
DEGREES OF
FREEDOM
MEAN
• SQUARE
F
Pretest Mean
.45844
1
.45844
(25.08)
PM at SS^o
.00832
1
.00832
<T.OO
PM at SS-jQQ
.75050
1
.75050
41.06*
Reliability
.17552
5
.03510
1.92
Sample Size
.33018
1
.33018
(18.06)
SS at PM-,
.00035
1
.00035
<1.00
SS at PM2
.63021
1
.63021
34.48*
PM X R
.15727
5
.03145
1.72
PM X SS
.30038
1
.30038
16.43*
R X SS
.10189
5
.02038
1.11
Residual
.09139
5
.01828
Total
1.61507
23
''Sample statistic greater than critical value at .05 level
38
TABLE 10
ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS
FROM THE STANDARD ANALYSIS OF COVARIANCE
AS THE CRITERION VARIABLES
SOURCE
SUM OF
SQUARES
DEGREES OF
FREEDOM
MEAN
SQUARE
F
Pretest Mean
.97768
1
.97768
(85.54)
PM at SS^o
.01620
1
.01620
1.42
PM at SS^QQ
1.61334
1
1.61334
141.15*
Reliability
.12431
5
.02486
2.17
Sample Size
.65076
1
.65076
(56.93)
SS at PM^
.00001
1
.00001
<1.00
SS at PM2
1 . 30482
1
1 . 30482
114.16*
PM X R
.19910
5
.03982
3.48
PM X SS
.65406
1
.65406
57.22*
R X SS
.05665
5
.01133
<1.00
Residual
.05714
5
.01143
Total
2.71970
23
*Saniple statistic greater than critical value at .05 level
39
TABLE 11
ANOVA SUMMARY TABLE USING MONTE CARLO GENERATED POWERS
FROM ANALYSIS OF COVARIANCE WITH PORTER'S ADJUSTMENT
AS THE CRITERION VARIABLES
SOURCE
SUM OF
SQUARES
DEGREES OF ,
FREEDOM
MEAN
SQUARE
F
Pretest Mean
.15714
1
.15714
(6.48)
PM at SS^Q
.00157
1
.00157
<1.00
PMatSS^OQ
.36018
1
.36018
14.85*
Reliability
.14179
5
.02836
1.17
Sample Size
.23285
1
.23285
(9.60)
SS at PH^
.00046
1
.00046
<1.00
SS at PM2
.43701
1
.43701
18.01*
PM X R
.16990
5
.03398
1.40
PM X SS
.20461
1
.20461
8.43*
R X SS
.13367
5
.02673
1.10
Residual
.12132
5
.02426
Total
1.16127
23
'^Sample statistic greater than critical value at .05 level
40
TABLE 12
COEFFICIENTS FOR THE LINEAR CONTRASTS USED WHEN
RELIABILITY WAS SIGNIFICANT IN THE ANALYSIS OF VARIANCE
LEVELS OF RELIABILITY
RELIABILITY OF GAIN GROUP
RELIABILITY OF NO GAIN GROUP
90
70
50
76
50
42
90
70
50
90
70
50
1
1
1
1
1
1
3
3
3
"3
"3
"■3
1
1
0
1
1
0
2"
"2"
Z
T
1
0
1
T
0
1
2
-T
2
"2
0
1
1
0
1
1
2
"2
2
"2
H-l
CONTRAST
CHAPTER V
DISCUSSION
In general, the results of this study support the positions of
Lord (1967, 1969), Campbell and Erlebacher (1970), and O'Connor (1972)
: with respect to their warnings about the implications of analysis of
covariance using unreliable test scores. The study shoves that even when
there is no gain in either group, a much higher fraction of rejections
occur than would be expected. The fraction of rejections is even more
extreme when the reliability is .70 or lower and the pretest means
differ. In addition. Porter's adjustment seems to offer little improve-
ment.
Comparison of the Two Methods of Analysis
The rejection of all forty-eight null hypotheses concerning the
equality of the sampling distributions for the two methods of analysis
shows that there is a difference in the results obtained from analysis '-'^
of covariance and analysis of covariance with Porter's adjustment. A
closer examination shows that these differences are more extreme when
the pretest means of the two groups differ and the reliabilities are ''
low. Further study shows that although the two methods of analysis are
different, neither method does an adequate job of modeling reality, that
is, both methods tend to produce erroneous proportions of type I and II
errors v/hen pretest means differ and the reliabilities are low. When the
pretest means are equal, the power of the tests seem to be directly re-
lated in a positive manner to the reliabilities when other variables are
held constant.
41
42
Possibly the most far reaching results were a function of re Ij Ability-.
The data indicated that incorrect decisions about which group had the
larger gain could be made using either method of analysis when the pre-
test means of the two groups differed and the reliabilities were low.
When the pretest mean of the gain group was| less than that of the no-gain
group, the adjusted posttest means stiowed that the no-gain group was
superior both when the mean gain was zero in both groups and when the
mean gain was positive only in the gain group. This possibility was
pointed out by Lord (1967) land Campbell and Erlebacher (1970).
■■■■-• I ■■'.-'
Factors that Affect Alpha and Beta
The factorial experiments using the computer generated alphas and
betas allow one to infer which factors affect the levels of type I and
type II errors. Using Monte Carlo generated alphas from the standard
analysis of covariance in a factorial experiment indicated that an inter-
action between pretest means and reliability and an interaction between
sample size and pretest means affected the alphas significantly. Further
analyses (simple effects) showed that the pretest mean factor was signif-
icant at every level of reliability. Both reliability and sample size
were significant when the pretest means differed. The tests of linear
contrasts showed that there was no significant difference between equal
reliabilities and unequal reliabilities when the pretest means differed,
but there were significant differences among the levels of reliability
when the pretest means differed. These results indicate that a difference
in reliabilities between groups, thus a difference in slopes, has no
effect, however, the level of reliability does.
The other three factorial experiments indicated that interactions
43
between pretest means and sample size were the major contributors to the
differing levels of alpha and power. In all three experiments (See Tables
10, 11, and 12) the sample size was significant when the pretest means
differed. Reliability seems to have a somewhat moderate effect in these
cases.
Predicting Alpha and Power
Using the results of this study, a regression equation can be set
up to predict the experimental probability of a type I error or the
experimental probability of rejecting a false null hypotheses when the
established probabilities are .05 and .50 respectively. The basic
regression equation is
(41) Y = eo+3TRG+32RN+B3S+34M+B5RQRf^+36RGS
+e7RQM+B3RfjS+3gR[^M+6iQSM+G
where •
Y = the predicted alpha or power,
Rq = the reliability of the gain group scores,
R^ = the reliability of the no gain group scores,
S = sample size
M = 1 if the pretest means are equal,
0 otherwise, and
3i = the regression coefficient, i.
Tables 14 and 15 provide confidence intervals for the expected
values of alpha and power respectively for each set of conditions noted.
44
TABLE 13
NINETY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE EXPECTED
VALUES OF ALPHAS UNDER SPECIFIED CONDITIONS
LOWER
UPPER
RELIABILITY
RELIABILITY
GROUP
EQUALITY
CONFIDENCE
CONFIDENCE
OF GAIN
OF NO GAIN
SAMPLE
OF PRETEST
LIMIT OF
LIMIT OF
GROUP SCORES
GROUP SCORES
SIZE
MEANS*
ALPHA
ALPHA
.90
.90
10
1
- .000
.137
.90
.90
10
0
.000
.137
.90
.90
100
1
.000
.100
.90
.90
100
0
.556
.741
.70
.70
10
1
.017
.162
.70
.70
10
0
. 1 08
.325
.70
.70
100
1
.000
.141
.70
.70
100
0 .
.780
.945
.50
.50
10
1
.000
.125
.50
.50
10
0
.297
.451
.50
.50
100
1
.000
.120
.50
■ .50
100
0
.932
1.000
.76
.90
10
1
.000
.136
.76
.90 .
10
0
.043
.223
.76
.90
100
1
.036
.143
.76
.90
100
0
. .691
.871
.60
.70
10
1
.003
.136
.60
.70
10
0
.228
.361
.60
.70
100
1
.014
.146
.60
.70
100
0
.879
1.000
.42
.50
10
1
.000
.097
.42
.50
10
0
.307
.472
.42
.50
100
1
.000
.117
.42
.50
100
0
.967
1.000
*If 1, the pretest means are equal and if 0, the pretest means
are not equal .
45
TABLE 14
NINETY-FIVE PERCENT CONFIDENCE INTERVALS FOR THE EXPECTED
VALUES OF POWER UNDER SPECIFIED CONDITIONS
LOWER
UPPER
RELIABILITY
RELIABILITY
GROUP
EQUALITY
CONFIDENCE
CONFIDENCE
OF GAIN
OF NO GAIN
SAMPLE
OF
PRETEST
LIMIT OF
LIMIT OF
GROUP SCORES
GROUP SCORES
SIZE
MEANS*
POWER
POWER
.90
.90
10
.000
.300
.90
.90
10
0
.000
.115
.90
.90
100
.000
.191
.90
.90
100
0
.351
.666
.70
.70
10
.019
.266
.70
.70
10
0
.048
.295
.70
.70
100
.000
.236
.70
.70
100
0
.677
.924
.50
.50
10
.000
.176
.50
.50
10
0
.155
.417
.50
.50
100
.000
. 224
.50
.50
100
0
.863
1.000
.76
.90
10
.000
.265
.76
.90
' 10
0
.000
.199
.76
.90
100
.000
.235
.76
.90
100
0
.523
.829
.60
.70
10
.000
.210
.60
.70
10
0
.097
.323
.60
.70
100
.010
.236
.60
.70
100
.782
1 .000
.42
.50
10
.000
.130
.42
.50
10
.156
.438
.42
.50
100
.000
1.000
.42
.50
100
0
.909
*If 1, the pretest means are equal and if 0, the pretest means
are not equal .
46
The predicted value of alpha or power can be found by using the
data found in Tables 4, 5, 6, and 7 to fit regression equation (41) to
obtain the following estimated regression parameters:
(42) alpha = .228 + .395Rq + .332R,^ + .008 S - .661M
-1.09 RqR,^ - .004 RqS + .619 RgM
-.003 R|^jS + .195 R|^M - .007 SM
and
(43) power = .045 + .536Rg + .404 R|^ + .010 S - .700 M
-1.228 RqR,^ - .006 RqS + .846 RgM
+ .002 RfjS + .218 RfjM - .007 SM.
A confidence interval for the predicted values of alpha and power
could be obtained in the usual manner.
A Direction for Further Research
The results of this study combined with other research (Lord, 1967;
Campbell and O'Connor, 1972), indicates a need for further study and the
development of a robust method for comparing groups of differing ability
when the scores are not perfectly reliable. Any new technique which is
proposed should first be investigated under known conditions, either
analytically, or by Monte Carlo techniques. This would insure that
another inappropriate method is not used for the evaluation of compensa-
tory education projects.
CHAPTER VI
SUMMARY
This study was designed to determine if either analysis of covar- ^.
iance or analysis of covariance with Porter's adjustment is an appropri-
ate analytical procedure for evaluating educational pretest-posttest
experiments. In particular, these methods: were compared with respect to
their use in the analysis of compensatory education projects where the
groups may differ in ability.
The study was carried out by computer generating 2000 sets of
normal data under forty-eight sets of predetermined conditions of reli-
ability, sample size, gain, and equality of pretest means. Each of the
2000 sets of data for each set of conditions was then analyzed using both
standard analysis of covariance and analysis of covariance with Porter's
adjustment. A sign test was used to compare the two methods of analysis
under each of the forty-eight sets of conditions. It was concluded that -^
the two methods of analysis yielded different results.
Factors affecting the two methods of analysis were then studied
separately using the computer generated alphas and powers as criterion
variables in four factorial experiments. The factors included reliability
at six levels, sample size at two levels, and the equality of pretest
means at two levels. From these experiments, it was concluded that pre- "^
test means interacting with sample size and sometimes with reliability
were significant factors. More specifically, sample size was statistically
significant in each case where the pretest means differed. Also, pretest
means were significant at ^M^r^ level of reliability for the computer
47
48
generated alphas produced by the standard analysis of covariance.
It was also learned that when the pretest means differ, both
standard analysis of covariance and analysis of covariance with Portier's
adjustment produced erroneous results with respect to which group if
either had a gain. When both groups had a mean gain of zero and the
pretest means differed, significant results usually indicated that the
group with the larger pretest mean had the gain. This would correspond
to the control group sampled from the general population being credited
with the gain in a compensatory education experiment. When there was a
gain in only one group and the pretest mean was lower in that group, the
analyses still indicated that the other group had the gain.
The results of this study point to the recommendations that analysis
of covariance with or without Porter's adjustment should be approached ^
with caution when the reliabilities are below .90 and the pretest means
(covariate means) are likely to be different for the groups.
APPENDIX
50
FCRTRAN PRCGRAN WHICH PERFORMED DATA GFNERATIOiN AND ANALYSIS
DIh'ENSICN EIGN(IOO), E2GN( 100) ,GAINGN{ ICO) tXX( 2G0 ) , YY ( 200 ) ,
»X(100) .ElNGdOO), E2NG( 100 ),GAINNC( I0a),TG(100) ,TN(I00)
REAL .NS[3,NSE,MSBP,MSeP
C
c
c
c
c
c
c
c
c
c
99
100
101
C
c
c
102
103
104
105
C
C
C
. NGRCUP=0
INPUT CF GRCUP PARAMETERS WHEREO
RELGN REPRESENTS THE RELIABILITY OF THE GAIN GROUP DATA
RELNG REPRESENTS T HE R EL I AB IL I T Y OF THE NO GAIN GROUP
NPERG REPRESENTS THE NUMBER OF OBSERVATIONS PER GROUP
GBAR REPRESENTS THE MEAN GAIN FOR GAIN GROUP
PR^NG REPRESENTS THE GAIN GROUP PRETEST MEAN
FR^NNG REPRESENTS THE NO GAIN GROUP PRETEST MEAN
ISEEC REPRESENTS THE SEED FOR RANDOM NUMBER GENERATOR
READ (5,99) I S EED , NGP , NS PL
FORMAT ( IH,2X,2I5)
READ 15,101) RELGN, RELNG, NPERG, GDAR,PRMNG,PRMNNG
FORMAT {2F3.2, 13, F3.2,2F3.0)
NGRCUP = NGl-iGUP + l
NSAh'F = 0
NF10=0
NF 100=0
NFP10=0
NFP100=0
HEADER CARD FOR NEW SET OF PARAMETERS
WRITE (6,102)
FORMAT (• l',T43, ' F STATISTICS BASED ON THE FOLLOWING PARME'
• ,«TERSM
WRITE (6,103)
FORMAT ( •0',T22, 'RELGN •,2X, 'RELNG •,2X, 'NPERG ',2X,' GBAR'
»,' 'i^X.'PRMNG • ,2X, 'PRMNNG' )
WRITE (6,104) RELGN, RELNG, NPERG, GBAR, PRMNGfPRMNNG
FORMAT (1X,T22,2F8.5, I5,3X,3F8.3)
WRITE (6,105)
FORMAT ( '0' ,T45, "SAMPLE NUMB ER' , 5X ,' STANDARD F ', 5X ,' PORTER '
», 'S F')
COMPUTE VARIANCE COMPONENTS
VARE1G=100»{1-RELGN)
VARE1N=100«(1-RELNG)
VARTG=100«RELGN
VARTK=100"RELNG
VARGNG=.04»VAf?TG
VARGNN=.04«VARTN
51
C
106
C
C
C
C
c
c
300
c
c
c
301
C
c
c
302
C
C
C
303
C
C
C
304
C
C
C
C
C
C
VARE2G=(V/iRTG+VARGNG)/RELGN-VARTG-yARGNG
VARE2N=( VARTN+VARGNN ) /RELNG-V ARTN-VARGNN
CCNTINUc
NSAMP=;NSA^P + 1
GENERATICN CF CATA FOR GAIN GROUP
TRUE SCORES
CALL RAKGENCNPERG, ISEECX)
DO 3C0 I=1,NPERG
TG(I )=X( I ) •SQRTIVARTGI+PRMNG
CCNTINUE
PRETEST ERROR SCORES
CALL RANGEN(NPERG,ISEED,X)
CO 301 I=1,NPERG
eiGN(I)=X(I)»SeRT(VARElG)
CCNTINUE
POSTTEST ERROR SCORES
CALL RANGEN(NPERG,.IS'EEn,,X)
DC 302 I^l.NPERG
E2GN( I)=XI I)»SCRT(VARE2G)
CONTINUE
GAIN SCORES
CALL RANGEMNPERG, ISEEO.X )
CO 303 ■ I = 1,NPERG
GAINGNi I )=X( I )«SQRT{ VARGNG)+GBAR
CONTINUE
PRETEST ANC POSTTEST SCORES
[JO 30 A I = 1,NPERG
XXd )=TG{ I)+E1GN( I )
YY( I l=TG{ I)+GAINGN( I ) + E2GN( I )
CCNTINUE
GENERATION OF CATA FOR NO GAIN GROUP
TRUE SCORES
CALL RANGEN(NPERG,ISEED,X)
00 400 I=1,NPERG
52
400
r.
c
c
401
c
c
c
402
C
C
c
403
C
C
C
404
C
C
C
C
C
C
C
TN( I )nX( I )«SQRT(VARTN)+PRMNNG
CCMINUC
PRETEST ERRCR SCORES
CALL PANGENCMPERG, ISEECX)
DC .401 I = 1,NPERG
. E1NG( I) = Xn )»SGRTIVAPE1N)
CCNTINUE
POSTTEST ERROR SCORES
CALL PANGEN(NPERG, ISEED.X )
CC 402 I=1,NPERG
E2NG( I)=X(I )»SCRT(VARE2N)
CCNTINUt
GAIN SCORES
CALL RANGEN(NPERG, ISEED.X)
DC 403 1=1,NPERG
GAINNG( I )=X( I )«SQRT{ VARGNN)
CCNTlKUt:
PRETEST ANC POSTTEST SCORES
DO 404 I=1,NPERG
L=NPERG+I
XX(L)=TN( I)+E1NG( I )
YY{L)=TN( I )+GAINNG{ I ) + E2NG( I )
CCNTINUE
STANCARC ANALYSIS OF COVARIANCE COMPUTATIONS
N=2*NPERG
IMTIALIZATICN
SUNXG=0.0
SUNX2G=0.C
SU^YG=0.0
SUMY2G=0.0
SUNXYG=O.C
SUNXN=a.O
50^X2^=0.0
SUN'YN=0.0
SUyY2N=0.C
SUNXYN=0.C
GROUP SUMS AND SUMS OF SQUARES
53
600
C
C
C
DC 6G0 I=1,NPERG
sur'XG=sur'XG+xx{i )
SUNX2G = SUNX2GtXX( I )»XX( I )
. SUr'YG = SUI^YG+YY( I )
SUNY2G = SUNY2G + YY( I )»YY( I )
. SU^'XYG = SU^XYG + XX( I )»YY( I )
K=NPtRG+I
SUr'XN^SUNXN+XXCK )
SU^X2N = SUr'X2N + XX(K)»XX(K)
SUNYN = SUNYN + YY (K )
SU^Y2^ = SU^■Y2^J+YY(K)»YY{K)
SUr'XYN=SUNXYrj+XX (K)»YY(K )
GCNTINUC
TOTAL SUMS AND SUMS OF SQUARES
TSUVX=SUNXG+SUMXN
TSUMX2=SU^X2G+SUMX2N
TSUI^Y = SU^'YG + SUMY^I
TSUNY2=SUNY2G+SUMY2N
TSU^XY=SUNXYG+SUMXYN
COMPUTE TOTAL SUMS OF SQUARES
CFX=TSUMX«TSU^X/N
CFY=TSUMY«TSU^Y/N
CFXY = TSUMX«TSU^'Y/N
TXX=TSUMX2-CFX
TYY=TSUKY2-CFY
TXY=TSU^XY-CFXY
COMPUTE BETWEEN GROUPS SUMS OF SQUARES
BXX= (SUMX'G*SUMXG4-SUMXN»SUMXN )/NPERG-CFX
BYY=(SUMYG«SUMYG+SUMYN*SUMYN)/NPERG-CFY
BXY= (SUMXG»SUMYG+SUMXN»SUMYN )/NPERG-CFXY
COMPUTE ERROR SUMS OF SQUARES
Exx=Txx-exx
EYY=TYY-eYY
EXY = TXY-e.XY
COMPUTE ACJUSTED SUMS OF SQUARES
TYYACJ=TYY-TXY«TXY/TXX
£YYACJ=EYY-EXY«EXY/EXX
BYYACJ=LiYY-{TXY»TXY)/TXX + (EXY«EXY)/EXX
54
C
G
80C
C
C
801
802
CCNPUTE ACJUSTEC MEAN SQUARES
^Se=BYYACJ/l.C
ySE = EYYA[:j/ (M-3)
CC^'PUTE F STATISTIC
. F = N'SB/MSE
ANALYSIS CF COVARIANCE WITH PORTER'S ADJUSTMENT
CCPPUTE CCRRELATION BETWEEN X AND Y
CENCN=TXX«TYY
RXY=TXY/SGRT( CENOy)
COMPUTE SUNS OF SQUARES WITH PORTER'S ADJUSTMENTS
EPCRT=EYYACJ
TPCRT=TYY-l {RXY»EXY+eXY)»«2)/( (RXY*RXY»EXX)+BXX)
BPCRT=TPCRT-EPCRT
COMPUTE MEAN SQUARES
NSEP=EPCRT/I.O
MSEP=EPCRT/{N-3)
COMPUTE F STATISTIC WITH PORTER'S ADJUSTMENT
FPORT-MSBP/MSEP
IF (F.GT.A.^5) NFlO=NFin+l
IF (F.GT.3.89J NF I00 = KF 1G0+, 1
IF (FFORT.GT.^.45) N FP 10 = NFP lC + 1
IF (FFCRT.GT.3.89) N FP IOO=NFP 1C0+ 1
WRITE CUT RESULTS
WRITE 16,800) NSAMP, F,FPORT
FORMAT (IX,T50, I4,T6A, F8.3,T77,F8.3)
IF (NSAMP. LT. NSPL ) GC TO 106
WRITE (6,801)
FORMAT ( 'C ,T10, •NF10.«,.T20, 'NF10C«,T30, 'NFPIO' ,T40, 'NFPIOO'
•)
WRITE (6,802) NF 10 , N F 100 , NFP IC ,NFP 100
FORMAT ( 1X,T10,4( H,6X ) )
IF (KGRCUP.LT.NGP) GC TO 100
STOP
END
55
SUDRCL'TINE RANGEN ( K, I R ,.X )
CINENSICN X(M)
1 = 1
CALL RANCU( IRtJR.Rl)
1R= JR
CALL PANCU( IR, JR,R2)
1R = JR
Rl = 2.G»<Rl-.5 )
R2 = 2.0«'{R2-.5 )
S2 = R1 «R1 + R2«R2
IF{S2.GT.1.0) GO TO 1
Y = SQRT(-2.0»( AL0G(S2)/S2) )
X(I)=(a«Y
IF(I.EG.N) GO to 2
X(Itl)=R2«Y
IF(I+1.EC.M) GO TO 2
1 = 1+2
GC TC 1
RETURN
END
SUBRCUTINE RANCU I IX , lY ,.Y FL )
I Y= IX •65539
IF(IY)5,6,6
5 IY=IY+2l^7^836A7+l
6 YFL=IY
YFL = Y(FL».^656613E-9
RETURN
END
BIBLIOGRAPHY
Box, G. E. P. and Muller, Mervin E. "A Note on the Generation of Random
Normal Deviates," Annals of Mathematical Statistics, 29 (June,
1958), 610-611. ' '
Campbell, Donald T. and Erlebacher, Albert, "How Regression Artifacts
in Quasi -Experimental Evaluations can Mistakenly Make Compen-
satory Education Look Harmful," in Disadvantaged Child, Vol. 3,
Ed. Hellmuth, Jerome. New York: Brunner/Marzel , Inc., 1970.
Campbell, Donald T. and Stanley, Julian C. Experimental and Quasi -
Experimental Designs for Research. Chicago: Rand McNally and
Company, 1963.
Cochran, William G. "Analysis of Covariance: Its Nature and Uses,"
Biometrics, 13 (September, 1957), 261-281.
Cronbach, Lee J. Essentials of Psychological Testing, Third Edition.
New York: Harper and Row, 1970.
Cronbach, Lee J. and Furby, Lita. "How We Should Measure 'Change' or
Should We?," Psychological Bulletin, 74 (January, 1970), 68-80.
DeCecco, John P. The Psychology of Learning and Instruction: Educa-
tional Psychology. Englev;ood Cliffs, N. J.: Prentice Hall, Inc.,
1968.
Ferguson, George A. Statistical Analysis in Psychology and Education,
Third Edition. New York: McGraw-Hill Book Company, 1971.
Fisher, Ronald A. Statistical Methods for Research Workers, Fourth
Edition. London: Oliver and Boyd Ltd., 1932.
Fisher, Ronald A. Statistical Methods for Research Workers, Tenth
Edition. London: Oliver and Boyd Ltd., 1946.
Fisher, Ronald A. The Design of Experiments. London: Oliver and Boyd
Ltd., 1935.
Garside, R. F. "The Regression of Gains Upon Initial Scores,"
Psychometri ka , 21 (March, 1956), 67-77.
Glass, Gene V. , Peckham, Percy D. , and Sanders, James R. "Consequences
of Failure to Meet Assumptions Underlying the Fixed Effects
Analysis of Variance and Covariance," Revjew of Educational
Research , 42 (Summer, 1972), 237-288.
56
57
Gulliksen, Harold. Theory of Mental Tests. New York: John Wiley and
Sons, Inc., 1950.
Helmstadter, G. C. Principles of Psychological Measurement. Nev; York:
Appleton-Century-Crofts, 1964.
Hicks, Charles R. "The Analysis of Covariance," Industrial Quality
Control , 20 (December, 1965), 282-287.
Hilgard, Ernest R. Theories of Learning. New York: Appleton-Century-
Crofts, Inc. , 1956.
Hill, Winfred F. Learning: A Survey of Psychological Interpretations,
Revised Edition. Scranton: Chandler Publishing Company, 1971.
Karmel , P. H. and Polasek, M. Applied Statistics for Economists, Third
Edition. Bath, Great Britain: Pitman Publishing, 1970. ~
Kirk, Roger E. Experimental Design: Procedures for the Behavioral
Sciences. Belmont California: Wadsworth Publishing Company,
Inc., 1968.
Lord, Frederic M. "A Paradox in the Interpretation of Group Comparisons,"
Psychological Bulletin, 68 (1967), 304-305.
Lord, Frederic M. "Elementary Models for Measuring Change," in Problems
JnMeasuring Change. Ed. Chester W. Harris. Madison, Wisconsin:
The University of Wisconsin Press, 1963.
Lord, Frederic M. "Further Problems in the Measurement of Growth,"
Educational and Psychological Measurement. XVIII (1958), 437-451.
Lord, Frederic M. "Large-Sample Covariance Analysis When the Control
Variable. is Fallible," Journal of the American Statistical
Association, 55 (1960), 307-321.
Lord, Frederic M. "Statistical Adjustments When Comparing Preexisting
Groups," Psychological Bulletin, 72 (1969), 336-337.
Lord, Frederic M. "Statistical Inferences about True Scores ," Psycho -
metrika. 24 (March, 1959), 1-17.
Lord, Frederic M. "The Measurement of Growth," Educational and Psycho-
logical Measurement, XVI (1956), 421-437.
Lord, Frederic M. and Novick, Melvin R. Statistical Theories of Mental
Test Scores. Reading, Massachusetts: Addi son-Wesley Publishing
Company, 1968 .
Manning, Winton H. and DuBois, Philip H. "Correlational Methods in
Research on Human Learning," Perceptional and Motor Skills.
15 (1962), 287-321.
58
Marks, Edmond and Martin, Charles G. "Further Comments Relating to the
Measurement of Change," American Educational Research Journal,
10 (Summer, 1973), 179-19T: ' ~
Marsaglia, G. and Bray, T. A. "A Convenient Method for Generating
Normal Variables," SIAM Review. 6 (July, 1964) .' 260-264.
McNemar, Quinn. "On Growth Measurement," Educational and Psychological
Measurement, XVIII (1958), 47-55.
Muller, Mervin E. "A Comparison of Methods for Generating Normal
Deviates on Digital Computers," Association for Computing .
Machinery Journal, 6 (1959), 376-383.
Neel , John Howard. A Comparative Analysis of Some Measures of Change.
Dissertation, University of Florida (1970).
O'Connor, Edward F., Jr. "Extending Classical Test Theory to the
Measurement of Change," Review of Educational Research, 42
(Winter, 1972), 73-97.
Peckham, Percy D. "The Robustness of the Analysis of Covariance to
Heterogeneous Regression Slopes," Paper read at the Annual
Meeting of the American Educational Research Association,
Minneapolis, Minnesota, (March 5, 1970).
Porter, Andrew Colvin. The Effects of Using Fallible Variables in the
' Analysis of Covariance. Dissertation, University of Wisconsin,
University Microfilms, Ann Arbor, Michigan:. (1967) .
Siegel , Sidney. Nonparametric Statistics for the Behavioral Sciences.
New York: McGraw-Hill Book Company, 1956.
Snedecor, George W. and Cochran, William G. Statistical Methods, Sixth
Edition. Ames, Iowa: The Iowa State University Press, 1967.
Thorndike, R., L. "Regression Fallacies in the Matched Groups Experiment,"
Psychdmetrika, 7 (June, 1942), 85-102.
Werts, Charles E. and Linn, Robert L. "A General Linear Model for
Studying Growth," Psychological Bulletin, 73 (1970), 17-22.
Winer, B. J. Statistical Principles in Experimental Design, Second
Edition. New York: McGraw-Hill Book Company, 1971.
BIOGRAPHICAL SKETCH
James Edwin McLean was born January 29, 1945, at Greensboro, North
Carolina. He grew up in Orlando, Florida and graduated from Edgewater
High School in June, 1963. He graduated from Orlando Junior College in,
January, 1966, and entered the United States Marine Corps Reserve.
In December, 1968, he received the degree. Bachelor of Science,
with a major in mathematics education from the University of Florida.
In January, 1969, he enrolled in the Department of Statistics at the
University and received the degree. Master of Statistics, in June, 1971.
During this period, he worked as a graduate assistant in that department
where he taught elementary statistics and probability. .
He accepted a teaching assistantship in the College of Education
at the University of Florida in September, 1971. He currently holds
that position part-time along with the position of research associate
for a Project Follow Through evaluation grant.
James Edwin McLean is a member of the American Educational Research
Association, National Council on Measurement in Education, the American
Statistical Association, and Phi Delta Kappa.
He is married to the former Sharon Elizabeth Robb and they have
no children.
59
I certify that I have read this study and that in my opinion it
conforms to acceptable standards of scholarly presentation and is fully
adequate, in scope and quality, as a dissertation for the degree of
Doctor of Philosophy.
William B. Ware, Chairman
Associate Professor of Education
I certify that I have read this study and that in my opinion it
conforms to acceptable standards of scholarly presentation and is fully
adequate, in scope and quality, as a dissertation for the degree of
Doctor of Philosophy.
A/''>'L^-(—-
VyXce A. Nines
Professor of Education
I certify that I have read this study and that in my opinion it
conforms to acceptable standards of scholarly presentation and is fully
adequate, in scope and quality, as a dissertation for the degree of
Doctor of Philosophy. [•
^Ji 1 1 laflrt'lftndenhal 1 i ~~
Profes~sorJ of Statistics
I certify that I have read this study and that in my opinion it
conforms to acceptable standards of scholarly presentation and is fully
adequate, in scope and quality, as a dissertation for the degree of
Doctor of Philosophy. ^ —
y ■ V ii
p. V. Rao
Professor of Statistics
I certify that I have read this study and that in my opinion it
conforms to acceptable standards of scholarly presentation and is fully
adequate, in scope and quality, as a dissertation for the degree of
Doctor of Philosophy.
n. ^m-^W
Jame^T. McClave
Assistant Professor of Statistics
This dissertation was submitted to the Dean of the College of Education
and to the Graduate Council, and was accepted as partial fulfillment of
the requirements for the degree of Doctor of Philosophy.
August, 1974
Dean, Graduate School
Live Music Archive
Librivox Free Audio
Metropolitan Museum
Cleveland Museum of Art
Internet Arcade
Console Living Room
Open Library
American Libraries
TV News
Understanding 9/11